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Abstract 


The  use  of  robots  by  the  U.S.  military  has  grown  exponentially  in  the  last  10  years. 
While  remotely  piloted  drones  and  ground  vehicles  are  already  in  the  spotlight  today,  the 
Department  of  Defense  (DoD)  has  stated  a  goal  of  increasing  the  level  of  automation  in 
unmanned  systems.1  Based  on  pre-existing  autonomous  systems,  the  DoD  goal,  and  ongoing 
technological  advances  in  artificial  intelligence  it  seems  likely  that  automated  lethal  robots 
will  soon  be  available  to  operational  commanders  for  use  in  combat.  While  autonomous 
lethal  robots  promise  significant  rewards,  they  also  bring  with  them  significant  risks  to  the 
success  of  military  operations.  Military  planners  should  choose  to  use  autonomous  lethal 
systems  when  the  importance  of  casualty  reduction  outweighs  the  importance  of  avoiding 
strategic  communications  setbacks  due  to  collateral  damage. 


Introduction 


In  2001,  Congress  passed  the  Floyd  D.  Spence  National  Defense  Authorization  Act 
(Public  Law  106-398)  which  mandated  two  milestones  for  military  robots,  “first,  that  by 
2010,  one  third  of  the  aircraft  in  the  operational  deep  strike  force  should  be  unmanned,  and 
second,  that  by  2015  one  third  of  the  Army’s  Future  Combat  Systems  operational  ground 
combat  vehicles  should  be  unmanned. Thus  was  born  the  ongoing  explosion  in  use  of 
unmanned  robotic  technology  by  the  U.S.  military.  Since  the  passage  of  Public  Law  106-398, 
more  than  6,000  Unmanned  Ground  Vehicles  (UGVs)  have  been  procured  and  deployed  to 
Iraq  and  Afghanistan. 

In  implementing  Congress’  mandate,  the  Department  of  Defense  published  the 
Unmanned  Systems  Roadmap ,  which  established  several  goals  including:  “Support  research 
and  development  activities  to  increase  the  level  of  automation  in  unmanned  systems  leading 
to  appropriate  levels  of  autonomy,  as  determined  by  the  Warfighter  for  each  specific 
platform.”4  Thus,  the  Unmanned  Systems  Roadmap  makes  development  and  employment  of 
automated  robots  a  goal  while  recognizing  that  it  is  not  appropriate  for  all  missions. 

However,  the  use  of  autonomous  lethal  robots  brings  additional  risks.  Robots  will 
make  errors  despite  the  best  intentions  and  design.  Much  like  the  humans  by  whom  they  are 
designed,  robots  can  be  spoofed  or  manipulated  and  can  make  mistakes.  Incidents  involving 
improper  application  of  lethal  force  by  robots  should  be  expected  to  have  a 
disproportionately  large  adverse  effect  on  the  strategic  communications  efforts  of  the 
operation.  Military  planners  should  choose  to  use  autonomous  lethal  systems  when  the 
importance  of  casualty  reduction  outweighs  the  importance  of  avoiding  strategic 
communications  setbacks  due  to  collateral  damage. 
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Why  is  this  operational? 


A  commonly  used  and  debated  phrase  among  military  professionals  is  “strategic 
corporal”.  This  phrase  highlights  that  important  decisions  are  being  made  at  the  junior  NCO 
level  and  these  decisions,  right  or  wrong,  can  have  strategic  consequences.  In  the  2010 
military  environment  where  counterinsurgency  efforts  demand  winning  the  hearts  and  minds 
of  the  population  and  a  camera  waits  around  every  corner,  the  mistakes  of  a  few  junior 
soldiers  can  have  huge  adverse  effects  on  the  success  of  the  operation. 

Enter  into  this  environment  a  new  breed  of  corporal,  tasked  to  make  these  same  sort 
of  decisions  that  can  have  strategic  consequences  if  the  decision  is  incorrect.  This  new 
corporal  is  an  autonomous  robot  that  has  been  programmed  with  artificial  intelligence  to 
make  decisions  about  if  and  when  lethal  force  can  be  used  without  human  interaction  (no 
man  in  the  loop).  And  this  corporal  is  armed  with  a  weapon  and  the  unflinching  readiness  to 
use  it.  If  this  new  breed  of  soldier  improperly  applies  lethal  force,  enemy  propaganda  and 
news  reporters  will  race  to  break  the  story  about  the  American  robot  that  went  rogue.  The 
negative  impact  of  such  mistakes  could  have  disastrous  consequences  for  the  strategic 
communications  effort  and  therefore  on  the  success  of  the  operation. 

Therefore  when  deciding  which  forces  are  to  be  used  to  pursue  an  objective, 
operational  planners  must  carefully  consider  the  risks  and  rewards  of  their  use  and  ensure 
that  they  are  choosing  the  right  forces  to  meet  their  objective.  Additionally  operational 
planners  must  ensure  that  decisions  about  the  employment  of  these  forces  are  made  at  the 
correct  level. 
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Background 


Use  of  robotic  systems  over  the  last  decades  has  proven  that  robots  can  significantly 
contribute  to  the  success  of  military  operations.  However,  the  DoD’s  Unmanned  System 
Roadmap  states  that,  .  .these  successes,  however,  likely  represent  only  a  fraction  of  what  is 
possible  and  desirable  by  employing  unmanned  systems.”5  As  the  DoD  looks  to  increase  its 
use  of  unmanned  robotics  systems  and  simultaneously  looks  to  increase  the  automation  of  the 
robots  in  use,  the  robotics  industry  is  churning  to  design,  develop,  and  produce  the  systems. 
This  has  led  to  what  one  source  describes  as  a  “technological  stampede  for  cool  stuff’  which 
unfortunately  leads  to  roboticists  designing  machines  from  a  technological  perspective  rather 
than  a  human  systems  or  mission  capability  perspective.6  The  Unmanned  Systems  Roadmap 
has  identified  the  rapid  pace  of  advancement  as  a  challenge  to  future  success,  stating  that, 

“the  development  of  unmanned  system  requirements  that  are  driven  by  what  is  demonstrated 
by  vendors  rather  than  vendors  developing  systems  based  on  DoD  requirements.”7  Thus  there 
is  an  acquisition  challenge  in  making  sure  requirements  are  driving  design  instead  of  vice 
versa. 


The  operationally  relevant  portion  of  this  challenge  is  understanding  that  the 
requirements  and  capabilities  of  these  systems  are  still  being  developed  and  that  the 
capabilities  of  these  systems  may  or  may  not  match  those  required  for  a  specific  operation. 
Therefore  operational  planners  must  carefully  weigh  the  risks  and  rewards  of  these  systems 
with  the  understanding  that  they  may  have  performed  very  well  in  some  roles  but  may  be 
poorly  suited  for  others.  The  Unmanned  Systems  Roadmap  states  that  the  appropriate  level  of 

Q 

autonomy  must  be  determined  by  the  warfighter,  and  this  paper  should  arm  the  planners  that 
support  the  warfighter  with  the  knowledge  to  make  such  determinations. 
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Definition  of  autonomous  lethal  robots 


In  order  to  frame  this  discussion  it  is  important,  though  not  easy,  to  define  the  terms 
being  used.  There  are  several  ways  to  define  the  word  “robot”.  While  some  definitions  use 
terms  like  “humanlike”  and  “operates  automatically”,  others  attempt  to  define  a  robot  by 
what  it  does.  Robotics  expert  and  defense  consultant  Peter  Singer  defines  a  robot  as  a 
machine  that  senses,  processes,  and  acts.9  Specifically  a  robot  must  be  able  to  sense  or  detect 
things  or  conditions  in  its  environment,  perform  some  sort  of  processing  to  determine  a 
reaction  to  that  condition,  and  then  use  some  sort  of  “effector”  to  alter  its  environment. 
Sensing  can  use  many  portions  of  the  electromagnetic  spectrum  (radar,  IR,  visible  light, 
laser)  to  detect  an  item  or  condition  of  interest.  Processing  could  entail  a  wide  range  of 
artificial  intelligence,  used  to  turn  the  sensory  inputs  into  a  command  to  use  an  “effector”. 
“Effectors”  are  anything  that  the  robot  uses  to  influence  its  environment,  from  a  searchlight 
to  a  robotic  claw  to  a  Hellfire  missile.  Thus  a  remote  controlled  airplane  would  not  be  a  robot 
as  it  has  no  sensors  and  no  capacity  to  make  decisions.  Conversely  a  remotely  piloted  drone 
(such  as  a  Predator),  even  though  it  defers  most  decisions  to  its  operator,  can  fly  itself 
automatically  while  “sensing”  and  “effecting”  militants  in  the  tribal  regions  of  Pakistan  and 
thus  could  be  defined  as  a  robot. 

Lethal  robots  are  those  armed  with  the  means  to  kill  humans.  This  would  include  both 
robots  specifically  designed  for  close  quarters  combat  against  humans  (armed  with  shotguns, 
machine  guns,  etc.)  and  those  with  a  kinetic  warhead  designed  for  destruction  of  enemy 
infrastructure  (drones  designed  to  find  and  destroy  enemy  air  defenses).  Robots  would  be 
non-lethal  if  armed  only  with  cameras  or  with  non  lethal  weapons  such  as  Tasers. 
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Finally  it  is  important  to  define  the  term  autonomy.  The  Department  of  Defense  has 
defined  10  distinct  levels  of  automation.10  Peter  Singer  describes  seven  distinct  levels  of 
automation  that  include  “direct  human  operation”,  “human  assisted”,  “fully  autonomous”, 
and  “adaptive.”11  Thus  a  remotely  piloted  vehicle  demonstrates  the  lowest  level  of  autonomy 
as  most  or  all  of  its  “processing”  is  done  by  its  human  controller.  This  means  that  the 
remotely  piloted  drone  missions  being  flown  today,  where  the  remote  operator  commands 
weapons  release,  would  be  a  “direct  human  intervention”  lethal  robot  as  opposed  to  an 
“autonomous  lethal  robot”.  With  increasing  levels  of  autonomy,  a  human  goes  from  a  direct 
operator  to  an  assigner  of  automated  tasks  to  an  observer  of  automated  activity.  It  is 
important  to  note  that  even  the  highest  levels  of  autonomy  have  human  interaction.  The 
iRobot  Scooba  floor  cleaner  is  an  example  of  an  autonomous  non-lethal  robot.  The  Scooba  is 
a  commercially  available  floor  cleaning  system  that  requires  a  human  to  designate  an  area  of 
operations  via  either  mechanical  limits  such  as  closed  doors  or  electronic  limits  such  as 
iRobot  “walls”  and  to  initiate  its  cleaning  cycle  by  pushing  a  button.  At  that  point  it  requires 
no  further  human  interaction  as  it  uses  a  preprogrammed  pattern  and  obstacle  recognition  to 
autonomously  clean  its  designated  area.  The  “fully  autonomous”  Harpy  drone,  one  of  the  few 
autonomous  lethal  robots  ready  for  use  today,  still  needs  human  direction.  Prior  to  launch,  it 
requires  human  input  to  determine  the  desired  area  of  operation  and  what  types  of  targets  it 
should  locate  and  destroy.  Post  launch  it  operates  in  a  fully  autonomous  mode  as  it  flies  to  its 
commanded  area  of  operations,  locates  its  target,  and  attacks  its  target.  For  the  purposes  of 
this  paper,  the  level  of  autonomy  in  question  is  whether  the  robot  has  been  given  the  decision 
to  use  lethal  force,  the  “decision  to  pull  the  trigger.” 
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Therefore  an  autonomous  lethal  robot  is  a  machine  that  can  sense  its  conditions,  can 


make  an  independent  (no  human  intervention)  decision  about  the  use  of  lethal  force,  and  then 
can  apply  that  lethal  force  using  its  own  effectors. 

Is  this  just  science  fiction? 

To  clarify  with  readers  that  such  systems  are  not  just  something  in  a  science  fiction 
movie,  autonomous  lethal  systems  exist  today  and  some  are  deployed  for  use  today.  Two 
older  examples  of  autonomous  lethal  systems  are  the  Phalanx  shipboard  self  defense  system 
and  the  Tomahawk  land  attack  missile.  The  Phalanx  is  capable  of  “autonomously  performing 
its  own  search,  detect,  evaluation,  track,  engage,  and  kill  assessment  functions.”  The 
Tomahawk  land  attack  cruise  missile  flies  a  preprogrammed  route  and  uses  digital  scene 
mapping  to  confirm  and  attack  its  target. 

More  recent  examples  of  autonomous  lethal  systems  with  more  robot-like  qualities 
are  the  Harpy  attack  drone  and  the  SGR-A1  security  guard  robot.  The  Harpy  is  a  small  aerial 
drone  produced  by  Israel  and  designed  to  provide  an  autonomous,  'fire-and-forget' 
Destruction  of  Enemy  Air  Defenses  (DEAD)  capability.  Harpy  flies  a  preprogrammed  route 
over  the  ground  using  its  onboard  passive  radar  receiver  to  detect  enemy  air  defense  targets 
and  then  flies  itself  into  the  target  where  it  detonates  its  32  kg  high-explosive  warhead.  It 
requires  no  man  in  the  loop  oversight  and  makes  its  own  decisions  about  what  constitutes  a 
valid  target.  The  SGR-A1  is  a  robotic  security  guard  produced  by  Samsung  and  deployed  in 
the  South  Korean  Demilitarized  Zone.  It  can  detect,  identify,  and  engage  targets  in  the 
Demilitarized  zone  in  either  a  semi-autonomous  mode  (which  requires  human  consent  to  use 
of  lethal  force)  or  autonomous  mode  (in  which  the  SGR-A1  makes  the  decision  regarding  the 
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use  of  lethal  force). 14  The  autonomous  capabilities  of  these  machines  establish  that 
autonomous  lethal  systems  have  arrived  and  demonstrate  the  urgency  of  the  discussion  about 
how  they  should  be  used. 

Rewards  of  autonomous  lethal  robots 

In  seeking  a  balance  of  risks  and  rewards,  we  must  consider  both  the  positive  and 
negative  aspects  of  employment  of  these  systems.  The  rewards  of  the  use  of  robots  have  been 
amply  demonstrated  in  Iraq  and  Afghanistan.  Several  books  and  articles  on  robotics  tell  the 
story  of  a  Navy  Chief  Petty  Officer  writing  a  letter  home  after  one  of  his  best  Explosive 
Ordnance  Disposal  (EOD)  technicians  was  killed  while  attempting  to  disarm  an  Improvised 
Explosive  Device  (IED)  in  Iraq.  The  Navy  Chief  was  relieved  to  be  sending  the  letter  to  a 
robotics  company  in  Boston  instead  of  a  farmhouse  in  Iowa.15  This  highlights  that  the  most 
important  reward  of  the  use  of  remotely  operated  or  autonomous  systems  is  the  reduced 
human  exposure  to  danger  and  reduced  risk  of  human  casualties. 

Remotely  operated  vehicles  and  robots  are  generally  used  to  perform  work  that  falls 
under  the  three  “Ds”-  dangerous,  dull,  dirty.16  Dangerous  work,  such  as  disarming  explosive 
devices  or  manning  fixed  checkpoints,  exposes  humans  to  potentially  deadly  forces.  Dirty 
work,  such  as  operating  in  areas  suspected  of  chemical  or  biological  contamination,  exposes 
humans  to  environmental  agents  that  are  unhealthy.  Dull  work,  such  as  persistent 
surveillance  of  a  static  position,  overwhelms  the  limits  of  human  attention.  In  these  areas,  the 
limits  of  human  physiology  put  humans  at  risk  of  mission  failure  or  bodily  harm.  In  these 
areas,  the  use  of  robots  has  blossomed. 
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One  reason  robots  enjoy  a  significant  advantage  over  humans  in  “dangerous”  work  is 
that  they  contain  no  emotional  requirement  for  self  preservation  and  the  associated  legal 
concept  of  self  defense.17  Specifically,  Peter  Singer  quotes  an  Army  robot  operator 
describing  his  thoughts  on  his  robot  under  fire:  “The  SWORDS  (Special  Weapons 
Observation  and  Reconnaissance  Detection  System)  doesn’t  care  when  it’s  being  shot  at. 
Indeed,  it  would  like  you  to  shoot  at  it.  That  way  we  can  identify  you  as  a  valid  target  and 

1  o 

engage  you  properly.”  Unlike  a  human,  who  would  very  rarely  “like”  to  get  shot  at,  a  robot 
has  no  instinctual  urge  or  legal  obligation  to  defend  itself.  Therefore  it  can  hold  fire  when 
getting  shot  at  and  even  when  getting  hit,  until  target  identification  is  certain. 

An  additional  facet  of  the  robot’s  lack  of  self  preservation  is  the  lack  of  short  term 
and  long  term  emotional  response  to  combat.  Robotics  consultant  Ronald  Arkin  provides  a 
significant  data  about  the  propensity  of  human  soldiers  to  commit  and  tolerate  battle  field 
atrocities.  He  then  details  some  adverse  emotional  conditions  that  cause  human  soldiers  to 
commit  these  atrocities  including  revenge  for  lost  brothers  in  arms,  lack  of  understanding  of 
the  Law  of  Armed  Conflict,  frustration,  and  “bloodthirst”  or  pleasure  derived  from  the  power 
of  killing.19  Arkin  goes  on  to  state  that  robots  and  their  artificial  intelligence  could  and 
should  be  free  of  these  human  emotional  reactions  to  the  horror  of  the  battlefield.”  A 
machine  with  no  emotions  feels  no  fear  and  no  anger  and  does  not  change  its  decision 
process  when  under  fire.  A  Pentagon  official  once  summed  up  robot  reaction  to  combat  by 
saying,  “They  don’t  get  hungry.  They’re  not  afraid.  They  don’t  forget  their  orders.  They 
don’t  get  care  if  the  guy  next  to  them  has  just  been  shot.  Will  they  do  a  better  job  than 
humans?  Yes.”21 
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Robots  are  capable  of  fulfilling  a  wide  variety  of  missions.  Their  most  important 
contribution  is  in  the  three  “Ds”  where  their  use  can  reduce  the  risk  of  human  exposure  to 
danger.  Their  lack  of  preservation  instinct  allows  them  to  do  things  that  humans  don’t  want 
to  do  and  commanders  don’t  want  their  subordinates  to  do.  Therefore  the  employment  of 
these  systems  seems  very  advantageous.  However  the  negative  aspects  of  their  use  must  also 
be  considered. 

Risks  of  au  tonomous  lethal  robots 

The  risk  of  the  use  autonomous  lethal  robots  in  combat  can  be  reworded  into  the 
question,  what  will  be  the  effect  of  a  robot  that  improperly  employs  lethal  force?  Despite  all 
the  advances  in  robotics  technology,  robots  are  not  and  will  never  be  perfect.  An  autonomous 
lethal  robot  in  combat  will  eventually  kill  the  wrong  guy,  either  a  friendly  or  a  civilian. 

The  history  of  human  interaction  with  automated  systems  indicates  that  automation 
can  reduce  mistakes  in  some  tasks,  but  mistakes  will  not  be  eliminated.  In  1988,  the  U.S.S. 
Vincennes,  a  guided  missile  cruiser,  deployed  to  the  Persian  Gulf.  It  was  armed  with  the 
AEGIS  weapon  system  that  could  automatically  detect,  track,  classify,  and  engage  airborne 
targets.  While  transiting  the  Strait  of  Hormuz,  the  AEGIS  system  detected  an  air  contact  that 
it  identified  as  an  Iranian  F-14.  In  its  semiautonomous  mode,  it  recommended  to  its  human 
operators  that  this  target  should  be  engaged  with  a  weapon.  The  crew  consented  to  the 
engagement  and  the  Vincennes  shot  down  an  Iranian  Airbus  passenger  aircraft.”  The  highly 
automated  target  classification  and  identification  system  made  the  wrong  call  and  identified  a 
passenger  plane  as  a  tactical  aircraft.  The  humans  failed  to  question  the  computer’s 
assessment  because  its  automation  was  so  advanced.  Together  the  automated  system  and  its 
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human  operators  shot  down  an  airliner.  More  recently,  in  2007  an  automated  South  African 
AAA  gun  experienced  a  software  glitch  during  a  live  lire  exercise  and  “the  rogue  gun  began 
firing  wildly”.  It  continued  to  do  so  until  it  emptied  its  magazine  of  500  rounds  of  37mm 
anti-aircraft  shells,  killing  9  and  wounding  14  others."  The  human  operators  were  unable  to 
control  the  gun  and  several  operators  were  killed  in  trying  to  shut  it  down.  Again,  the 
automated  system  made  an  unpredictable  decision  to  open  fire.  These  are  two  examples  of 
automation  that  hasn’t  worked  as  planned  and  resulted  in  improper  employment  of  lethal 
force.  Neither  one  would  have  been  expected  by  its  designer,  its  operator,  or  its  commander. 
Peter  Singer  notes  a  survey  in  which  4%  of  American  factories  where  robots  are  present  have 
had  a  “major  robotic  accident.”"  Additionally,  he  puts  a  robotic  twist  on  Murphy’s  law  when 
he  states  that  the  “dark  irony  is  that  the  more  advanced  robots  get  the  more  complex  they 
become,  and  the  more  potential  they  have  for  a  failure. . .” 

This  potential  for  failure  can  be  compared  to  the  Clausewitzian  concept  of  “fog  of 
war”  and  “friction.”26  This  “friction”  could  be  due  to  internal  factors  where  small  coding 
errors  in  the  software  can  have  drastic  unintended  results  (reference  the  South  African  AAA 
gun)  or  to  external  factors  on  the  battlefield  such  as  electromagnetic  interference.  Potential 
sources  of  electromagnetic  interference  include  everyday  signals  such  as  cell  phones  and 
Wifi,  intentional  friendly  jamming  of  IED  detonation  signals,  or  enemy  jamming  of  robot 
datalink.  When  robbed  of  its  interaction  with  its  human  overseer  and  exposed  to 
electromagnetic  signals  for  which  is  was  not  designed,  it  is  difficult  to  tell  what  the  robots 
will  do.  Peter  Singer  quoted  operators  of  three  different  robots  in  use  today  who  said  that 
they  have  lost  control  of  their  robots  due  to  electromagnetic  interference.  While  some  robots 
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might  just  stop,  others  “drive  off  the  road,  come  back  at  you,  spin  around,  stuff  like  that.” 


10 


Beyond  just  losing  friendly  control  of  robots,  a  technologically  advanced  enemy 
might  even  attempt  to  hack  or  hijack  our  robotic  warriors.  Our  current  enemies  have  proven 
to  be  very  adaptable  and  have  developed  simple  counters  to  our  battlefield  tactics  and 
technology.  In  December  2009,  the  Wall  Street  Journal  reported  that  insurgents  in  Iraq  had 
hijacked  the  video  feed  from  a  Predator  drone.  Using  a  $26  piece  of  commercial  software, 
they  were  able  to  intercept  and  watch  video  downlink  being  used  by  operational  and  tactical 
commanders.  Luckily  these  insurgents  were  only  using  the  video  feeds  to  avoid  military 
operations.  But  their  ability  to  intercept  robot  datalink  demonstrates  the  ease  with  which 
robot  command  and  control  might  be  hacked.  It  is  reasonable  to  suspect  that  a  future  threat 
with  a  more  robust  cyber  capability  would  be  able  to  do  much  more  than  intercept  video 
datalink.  Additionally,  insurgents  have  already  demonstrated  that  they  can  capture  our  robots 
and  even  use  them  against  us.  One  American  counter- IED  robot  was  captured  and  was  later 
found  in  pieces  at  the  site  of  a  bomb  blast.  Insurgents  had  re- wired  the  robot  and  turned  it 
into  a  mobile  IED.  This  indicates  that  our  current  enemy  knows  that  our  machinery  can  be 
turned  against  us.  The  dangerous  next  step  is  the  adverse  propaganda  that  our  enemy  might 
gain  by  using  a  captured  robot  against  civilians.  It  could  be  difficult  for  us  to  win  hearts  and 
minds  if  an  American  robot  is  seen  killing  people  at  a  marketplace,  even  if  we  claim  that  it 
was  not  under  our  control  at  the  time. 

Finally,  enemy  forces  may  be  able  to  spoof  or  deceive  autonomous  systems,  inducing 
a  robot  response  to  suit  their  propaganda  purposes.  In  Fast  Forward  to  the  Robot  Dilemma , 
David  Bigelow  discusses  the  propaganda  implications  of  a  fictitious  scenario  in  which 
insurgents  induce  a  robot  to  shoot  and  stage  a  martyr  to  jump  into  the  robot’s  field  of  fire  at 
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the  last  minute.  The  insurgents  set  up  the  engagement  to  ensure  that  it  appeared  that  a  robot 
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had  killed  a  civilian  bystander.  In  the  article  a  thorough  investigation  eventually  exposed  the 
planned  martyrdom,  but  the  initial  uproar  had  the  United  Nations  investigating  the  killings 
and  all  robots  were  sidelined.  A  different  hypothetical  scenario  by  which  enemies  might  be 
able  to  spoof  a  robot  would  be  a  technologically  advanced  enemy  who  knew  the  relatively 
simple  design  of  the  Harpy  drone.  Armed  with  the  knowledge  that  it  senses  and  guides  to 
radar  energy,  a  crafty  enemy  might  attach  radar  emitters  to  hospitals  and  school  buses  in  an 
attempt  to  induce  a  Harpy  to  go  after  an  off  limits  target,  providing  a  “killer  robot”  news  clip. 
Finally,  insurgents  commonly  use  human  shields.  Peter  Singer  relays  a  story  from  a  Ranger 
in  Somalia  who  watched  “a  gunman  shoot  at  him  with  an  AK-47  that  was  propped  between 
the  legs  of  two  kneeling  women,  while  four  children  sat  on  the  gunman’s  back.  The  Somali 
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warrior  had  literally  created  a  living  suit  of  noncombatant  armor.”  Human  shields,  radar 
spoofing,  and  martyrdom  are  all  means  that  an  enemy  might  employ  to  induce  a  robot  to  kill 
apparently  innocent  bystander. 

There  are  a  concerning  number  of  ways  in  which  autonomous  lethal  robots  could  kill 
noncombatants  or  even  friendly  forces.  Since  humans  have  been  making  such  mistakes  in 
war  for  thousands  of  years,  it  is  worth  discussing  the  difference  in  public  perception  (and 
therefore  propaganda  effect)  for  a  human  and  an  autonomous  lethal  robot. 

Many  of  our  past  and  present  enemies  have  been  masters  of  propaganda,  besting  us  in 
strategic  communications  even  if  they  could  not  best  us  on  the  battlefield.  When  performing 
counterinsurgency  operations,  strategic  communications  to  win  the  hearts  and  minds  of  the 
population  and  bolster  the  popular  support  among  U.S.  and  coalition  nations  are  a  critical 
component  of  success.  Some  examples  of  military  actions  that  were  huge  strategic 
communications  setbacks  were  the  Mai  Lai  massacre  in  Vietnam  and  the  Abu  Ghraib  prison 
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situation  in  Iraq.  In  both  cases,  poor  decisions  were  translated  into  public  outrage  and  eroded 
public  support  for  continuing  operations.  The  new  facet  of  the  strategic  communications 
battle  is  the  introduction  of  killer  robots  to  the  battlefield.  Some  experts  say  use  of  robots  will 
make  us  look  like  we  turned  loose  “Terminator”  robots,  others  say  they  will  be  regarded  as 
monsters  not  weapons,  and  some  say  that  they  will  be  interpreted  as  a  signal  of  casualty 
aversion  which  will  encourage  terrorism  instead  of  demonstrating  military  strength.  ~  A 
survey  conducted  by  Ronald  Arkin  demonstrated  that  public  perception  does  not  find  the  use 
of  lethal  force  by  autonomous  systems  acceptable  and  that  the  respondents’  main  concern 
was  the  risk  of  civilian  casualties. 

The  history  of  robots  and  automated  systems  indicates  that  if  autonomous  lethal 
robots  are  used  in  combat,  they  will  eventually  improperly  employ  lethal  force.  After 
factoring  in  the  media  savvy  of  the  current  enemy  and  the  adversity  of  public  opinion  to 
robots  killing  humans,  it  seems  likely  that  the  improper  employment  of  lethal  force  by  a 
robot  will  have  huge  adverse  strategic  communications  impact  which  could  endanger  the 
mission. 

Balance  of  risks  and  rewards 

Operational  planners  and  commanders  must  balance  the  risks  and  rewards  of 
employing  these  systems.  The  decision  boils  down  to  acceptable  risks.  Specifically,  planners 
and  commanders  must  weigh  the  benefits  of  reduction  in  U.S.  and  coalition  military 
casualties  against  the  strategic  communications  impact  of  collateral  damage  to  non- 
combatants.  For  total  war  where  strategic  communications  may  be  less  of  a  consideration,  the 
risks  of  the  use  of  autonomous  systems  is  low.  The  mistakes,  hacking  or  spoofing  would  be 


13 


considered  acceptable  collateral  damage  while  the  reduction  in  human  exposure  to  danger 
would  be  a  powerful  reward. 

However  for  operations  where  total  means  are  not  used  and  avoiding  collateral 
damage  is  more  important  than  avoiding  human  casualties,  the  risks  are  much  higher.  The 
ability  to  limit  human  casualties  might  provide  a  means  to  extend  public  support  for  a 
lengthy  effort,34  but  the  risk  of  propaganda  and  strategic  communications  setbacks  could 
outweigh  the  benefits.  When  performing  counterinsurgency,  maintaining  a  human  in  the  loop 
provides  the  best  balance  of  minimizing  human  casualties  while  minimizing  risk  of  providing 
the  enemy  with  propaganda  fodder. 

Counterargument 

Some  would  argue  that  autonomous  lethal  robots  are  preferable  to  human  soldiers  for 
all  types  of  combat  operations.  They  would  point  to  the  lack  of  human  emotions  that  can 
cloud  battlefield  judgement,  the  ability  to  precisely  program  in  rules  of  engagement  into  the 
robots  artificial  intelligence,  and  past  human  responses  to  advances  in  technology. 

As  previously  discussed,  human  emotions  such  as  anger  and  fear  are  often 
encountered  in  battle  and  have  been  shown  to  cloud  judgement  and  contribute  to  battlefield 
atrocities.  Robotics  engineers  would  correctly  point  out  that  these  emotions  are  not  present  in 
their  machines  and  therefore  they  can  make  very  calculated  decisions  in  the  heat  of  combat. 

Robotics  engineers  would  also  point  out  that  they  expect  to  be  able  to  program  rules 
of  engagement  into  the  artificial  intelligence  of  their  robots.  Robot  consultant  Ronald  Arkin’s 
book  Governing  Lethal  Behavior  in  Autonomous  Robots  provides  guidance  for  programming 
an  “ethical  governor”,  a  portion  of  AI  that  incorporates  Rules  of  Engagement  and  the  Law  of 
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Armed  Conflict  into  mission  tasking  and  sensory  inputs.  If  it  operates  as  designed,  it  would 
ensure  that  all  actions  taken  by  the  robot  would  be  within  legal  parameters.  Other  sources 
discuss  how  emotions  such  as  guilt,  empathy,  joy,  or  sadness  might  be  designed  into  a 
machines  AI  to  make  it  behave  in  a  more  human-like  manner.36  This  could  give  the  robot 
some  sense  of  morality  in  addition  to  the  legality  proposed  by  Arkin. 

From  a  historical  context,  it  could  also  be  argued  the  aversion  to  use  of  robots  on  the 
battlefield  is  just  the  latest  version  of  resistance  to  new  technology.  It  has  been  likened  to  the 
introduction  of  the  tank  in  World  War  I  which  was  described  as  “a  weapon  of  terror,  like  a 
large  monster,  not  a  weapon.”  It  could  be  argued  that  human  response  to  use  of  robots  is  no 
different  than  that  response  to  when  humans  first  saw  guns,  trains,  cars,  or  airplanes.  The 
follow  on  to  this  argument  is  that  in  50  years  we  will  be  kicking  ourselves  for  not  jumping  on 
the  automated  robot  train  sooner. 

However,  the  technological  ability  to  bypass  emotion  and  program  rules  of 
engagement  may  reduce  but  does  not  eliminate  the  chances  of  improperly  applied  lethal 
force.  The  Director  of  the  Humans  and  Automation  Lab  at  MIT  has  forecast  the  future  of 
robotics  and  automation,  “...we  are  going  to  see  a  lot  of  cool  technology  in  various  forms  of 
research  phases,  but  we’ll  also  see  a  lot  of  failure  because  many  systems  won’t  be  designed 
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with  the  human  in  mind.”  Additionally  many  robotics  experts  seem  to  fail  to  grasp  the 
military  consequences  of  robotic  incidents.  While  American  industry  may  be  willing  to 
accept  a  4%  “major  robotics  mishap”  rate,  that  same  rate  of  failure  may  not  be  acceptable  to 
an  operational  commander  if  each  robotics  incident  induces  investigations,  additional 
oversight,  and  a  strategic  communications  setback.  While  a  “beginning  dump  of  physical 
memory”  screen  on  a  PC  is  usually  just  a  temporary  inconvenience,  a  software  or  hardware 
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glitch  on  an  autonomous  lethal  robot  has  the  capacity  to  turn  a  tactical  operation  into  an 
operational  or  strategic  disaster. 

Conclusion 

Autonomous  lethal  robots  are  coming  soon.  They  have  been  pushing  their  way 
toward  the  battlefield  for  years  and  the  push  will  continue  as  congressional  mandates  and 
industry  drive  us  toward  increased  automation.  As  operational  planners  we  must  understand 
the  risks  and  reward  associated  with  the  use  of  these  systems  in  order  to  make  sure  we 
choose  to  use  them  for  the  right  operations.  They  are  very  capable  but  bring  with  them  a  risk 
of  operational  setback  due  to  the  volatile  nature  of  public  opinion  on  their  use.  Total  war 
seems  to  be  an  appropriate  time  to  use  them  while  counterinsurgency  does  not  appear  to  be 
well  suited  for  their  use.  In  all  cases  operational  commanders  will  want  to  maintain 
authorization  for  their  use  at  the  operational  or  strategic  level. 

Recommendations 

o  Operational  planners  should  continue  to  use  robots  for  “3-Ds”  work 
o  Operational  planners  should  view  autonomous  lethal  robots  as  one  of  many  forces 
available  for  an  operation  and  expect  that  their  use  is  not  appropriate  for  all 
operations 

o  Operational  planners  should  avoid  use  of  autonomous  lethal  robots  when  military 
casualties  are  more  acceptable  than  collateral  damage.  Consider  use  of  non-lethal 
(Tasers  in  place  of  guns)  or  “man  in  the  loop”  systems  to  reduce  the  risk  of  a  strategic 
communications  setback  of  robots  killing  civilians. 
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o  Operational  planners  should  consider  use  of  fully  autonomous  systems  when 
collateral  damage  is  more  acceptable  than  casualties, 
o  Operational  planners  should  clarify  with  the  operational  commander  his  intent  for 
authorization  of  the  use  of  autonomous  lethal  robots  (who  can  authorize  their  use). 


17 


End  Notes 


(All  notes  appear  in  shortened  form.  For  full  details,  see  the  appropriate  entry  in  the  bibliography.) 

1.  U.S.  Department  of  Defense,  Unmanned  System  Roadmap ,  4. 

2.  Ibid,  5. 

3.  Ibid,  3. 

4.  Ibid,  xiv. 

5.  U.S.  Department  of  Defense,  Unmanned  System  Roadmap ,  1. 

6.  Tucker,  “U.S.  Deploys  Unmanned  Vehicles.” 

7.  U.S.  Department  of  Defense,  Unmanned  System  Roadmap ,  40. 

8.  Ibid,  xiv. 

9.  Singer,  Wired  for  war,  67. 

10.  Valavanis,  Advances  in  unmanned  aerial  vehicles,  537-538. 

11.  Singer,  Wired  for  war,  74. 

12.  Arkin,  Governing  lethal  behavior  in  autonomous  robots,  7-8. 

13.  Streetly,  Jane ’s  Electronic  Mission  Aircraft,  313-315. 

14.  Arkin,  Governing  lethal  behavior  in  autonomous  robots,  10. 

15.  Singer,  Wired  for  war,  21. 

16.  Ibid,  63. 

17.  Arkin,  Governing  lethal  behavior,  46. 

18.  Singer,  Wired  for  war,  31. 

19.  Arkin,  Governing  lethal  behavior,  33. 

20.  Ibid,  36. 


18 


21.  Weiner,  "New  Model  Army  Soldier". 

22.  Arkin,  Governing  lethal  behavior,  30. 

23.  Singer,  Wired  for  war,  125. 

24.  Ibid,  196. 

25.  Ibid,  195. 

26.  Ibid,  195. 

27.  Ibid,  196. 

28.  Gormon,  "Insurgents  Hack  U.S.  Drones". 

29.  Singer,  Wired  for  war,  219. 

30.  Bigelow,  "Fast  Forward  to  Robot  Dilemma,"  18. 

31.  Singer,  Wired  for  war,  402. 

32.  Ibid,  311-313. 

33.  Arkin,  Governing  lethal  behavior,  53-55. 

34.  Singer,  Wired  for  war,  221. 

35.  Arkin,  Governing  lethal  behavior,  127. 

36.  Wallach,  Moral  machines,  155-159. 

37.  Arkin,  Governing  lethal  behavior,  312. 

38.  Ibid,  30. 

39.  Tucker,  “U.S.  Deploys  Unmanned  Vehicles.” 


19 


Bibliography 


Arkin,  Ronald  C.  Governing  lethal  behavior  in  autonomous  robots.  Boca  Raton,  FL:  CRC 
Press,  2009. 

Bigelow,  David  F.  "Fast  Forward  to  the  Robot  Dilemma."  Armed  Forces  Journal  (November 
1,2007):  18. 

Gormon,  Siobhan,  Yochi  Dreazen,  and  August  Cole.  "Insurgents  Hack  U.S.  Drones."  Wall 
Street  Journal,  December  17,  2009. 

Singer,  P.  W.  Wired  for  war:  the  robotics  revolution  and  conflict  in  the  twenty -first  century. 
New  York,  NY:  Penguin  Press,  2009. 

Streetly,  Martin.  Jane ’s  Electronic  Mission  Aircraft  Issue  24.  United  States:  Odyssey  press, 
2009. 

Tucker,  Patrick.  "U.S.  Deploys  Unmanned  Vehicles."  The  Futurist  43,  no.  6 
(November/December  2009):  12-13. 

U.  S.  Department  of  Defense.  Unmanned  Systems  Integrated  Roadmap  FY2009-2034. 

Washington  D.C.:  Office  of  the  Secretary  of  Defense,  April  2009. 

Valavanis,  K.  Advances  in  unmanned  aerial  vehicles:  state  of  the  art  and  the  road  to 
autonomy.  Dordrecht:  Springer,  2007. 

Wallach,  Wendell,  and  Colin  Allen.  Moral  machines:  teaching  robots  right  from  wrong. 
Oxford:  Oxford  University  Press,  2009. 

Weiner,  Tim.  "New  Model  Army  Soldier  Rolls  Closer  to  Battle."  New  York  Times,  February 
16,  2005. 


20 


Warrick,  Joby  and  Peter  Finn.  “In  Pakistan,  CIA  refines  methods  to  reduce  civilian  deaths; 
Smaller  missiles  New  precision  appears  to  dampen  outrage.”  Washington  Post ,  April 
26,  2010 


21 


